Please read https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout. Specifically, note that making the timeout higher sets both detection of timeout and the amount of time the server is considered offline before retrying connection.

@stefanv We have issues with the permissions checking causing the plotting to timeout (and some reasonable queries also being a bit on the slower side).

All I'm saying is that this may not be the silver bullet you need, since while it will tolerate longer timeouts, it will also cause nodes to be offline for longer.

If possible, you can make the slow endpoints async or add more endpoints (might require more CPUs).

@stefanv It's true neither the source nor the plot handlers are async currently. Do you want to try that first @bfhealy ?

@mcoughlin Sure, I'll take a look at making that change.